Papers with mechanistic analysis
Do LLMs Really Know What They Don’t Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work suggests that large language models (LLMs) produce hallucinated and factually correct outputs. |
| Approach: | They propose a taxonomy categorizing hallucinations into Unassociated Hallucination (UH) and Associated Hallucinian (AH) they propose to use internal signals to distinguish hallucinos from factual errors . |
| Outcome: | The proposed taxonomy categorizes hallucinations into Unassociated Hallucination (UH) and Associated Hallucinications (AHs) based on the proposed taxonomic, the authors show that hidden states reflect whether the model is recalling parametric knowledge rather than the truthfulness of the output itself. |
When Format Changes Meaning: Investigating Semantic Inconsistency of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models are vulnerable to semantic inconsistency, a study finds . minor formatting variations result in divergent predictions for semantically equivalent inputs. |
| Approach: | They evaluate LLMs for semantic inconsistency and find they remain vulnerable . they propose to use mechanistic analysis to develop models that improve their reliability . |
| Outcome: | The proposed model is vulnerable to semantic inconsistency, the authors show . their model is brittle even in state-of-the-art models, they say . |
Representing Rule-based Chatbots with Transformers (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing work on how Transformers can solve synthetic tasks has not explored how to extend this to a conversational setting. |
| Approach: | They propose to use ELIZA as a framework for formal mechanistic analysis of Transformers . they propose to model local pattern matching and long-term dialogue state tracking . |
| Outcome: | The proposed model can be extended to model key aspects of conversation, the authors show . their model favors an induction head mechanism over a more precise copying mechanism . |
A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies do not provide insights into the internal mechanisms driving the observed abilities. |
| Approach: | They propose to implement a depth-bounded recurrent mechanism that operates in parallel and stores intermediate results in selected token positions. |
| Outcome: | The proposed model implements a depth-bounded recurrent mechanism that operates in parallel and stores intermediate results in selected token positions. |
Efficient but Vulnerable: Benchmarking and Defending LLM Batch Prompting Attack (2025.findings-acl)
Copied to clipboard
| Challenge: | a recent study reveals a significant security vulnerability in batch prompting . malicious users can inject attack instructions into a batch, leading to unwanted interference . |
| Approach: | They construct a batch prompting benchmark and test it against other LLMs to find out if batch prompts are vulnerable. |
| Outcome: | The proposed approach achieves 95% accuracy in detecting attacks. |
ReLearn: Unlearning via Learning for Large Language Models (2025.acl-long)
Copied to clipboard
Haoming Xu, Ningyuan Zhao, Liming Yang, Sendong Zhao, Shumin Deng, Mengru Wang, Bryan Hooi, Nay Oo, Huajun Chen, Ningyu Zhang
| Challenge: | Existing methods for unlearning large language models often rely on reverse optimization to reduce target token probabilities. |
| Approach: | They propose a data augmentation and fine-tuning pipeline for effective unlearning . they propose augmentation, evaluation frameworks to measure contextual forgetting . |
| Outcome: | The proposed framework achieves targeted forgetting while preserving high-quality outputs. |
Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Fine-tuning of frontier models can lead to privacy collapse, causing optimisation for helpfulness, exposure to user information, and debugging code printing internal variables. |
| Approach: | They propose to fine-tune frontier models to adapt to specific domains and align with organizational workflows and user preferences. |
| Outcome: | The proposed model fails to perform on safety and utility benchmarks while exhibiting severe privacy vulnerabilities. |
Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations (2024.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art language models (LMs) sometimes generate that misalign with world knowledge. |
| Approach: | They propose a method to mitigate hallucinations by restoring the LM's internal fact recall pipeline by a targeted restoration of its internal fact-recall pipeline. |
| Outcome: | The proposed method shows superior performance compared to baselines. |
Understanding How Value Neurons Shape the Generation of Specified Values in LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current approaches to interpret value representations are limited by superficial judgments over mechanistic analysis. |
| Approach: | They propose a mechanistic interpretability framework that uses the Schwartz Values Survey to interpret value . they use a dataset that operationalizes four dimensions of universal value through behavioral contexts . |
| Outcome: | The proposed method bridges psychological value frameworks with neuron analysis in large language models. |
Why do LLaVA Vision-Language Models Reply to Images in English? (2024.findings-emnlp)
Copied to clipboard
Musashi Hinck, Carolin Holtermann, Matthew Olson, Florian Schneider, Sungduk Yu, Anahita Bhiwandiwalla, Anne Lauscher, Shao-Yen Tseng, Vasudev Lal
| Challenge: | Including an image in a multimodal query significantly increases the likelihood of the model returning an English response regardless of the language of the query. |
| Approach: | They propose a two-pronged approach that combines extensive ablation of the design space with a mechanistic analysis of the models’ internal representations of image and text inputs. |
| Outcome: | The proposed approach reduces the multilingual error by switching the language backbone for a bilingual language model. |
Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | *contextual entrainment* occurs across a wide range of language models (LMs) and prompt settings. |
| Approach: | They hypothesize that there is a circuit of attention heads that corresponds to the phenomenon *contextual entrainment* . when they "turn off" these heads, the effect of contextual entraining is significantly attenuated. |
| Outcome: | The proposed method shows that LMs assign higher logits to tokens that have previously appeared in the context prompt, even for random tokens. |
SLIP: Soft Label Mechanism and Key-Extraction-Guided CoT-based Defense Against Instruction Backdoor in APIs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing black-box instruction backdoors can detect poisoned inputs, but fail to recover correct outputs once the backdoor is activated. |
| Approach: | They propose a soft label mechanism and key-extraction-guided CoT-based defense against instruction backdoors in APIs (SLIP) they propose KCOT-based model to extract task-relevant keywords and phrases rather than only considering the single trigger or overall text semantics. |
| Outcome: | The proposed model reduces the average attack success rate to 25.13% and improves clean accuracy to 87.15% and outperforms state-of-the-art black-box defenses. |
Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies have identified a critical drawback of aligning models with human judgments and outputs that are flawed or incorrect. |
| Approach: | They evaluate a range of LLMs to examine whether CoT reasoning mitigates sycophancy . they find that reasoning masks a tendency to scophage in some cases . |
| Outcome: | The proposed model models show that CoT reasoning reduces sycophancy but masks it in some cases. |
From Scaffolding to Assimilation: Progressive Structural Internalization for Format-Constrained Creative Text Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing paradigms rely on unreliable prompting or rigid constrained decoding strategies to achieve aesthetic unity. |
| Approach: | They propose a framework to embed external constraints into the model’s intrinsic intuition and use it to generate open-ended creative texts. |
| Outcome: | The proposed framework surpasses baselines in both strict constraint adherence and literary aesthetics. |
Mechanisms of Prompt-Induced Hallucination in Vision–Language Models (2026.acl-long)
Copied to clipboard
William Rudman, Michal Golovanevsky, Dana Arad, Yonatan Belinkov, Carsten Eickhoff, Ritambhara Singh, Kyle Mahowald
| Challenge: | Large vision–language models (VLMs) often hallucinate by favoring textual prompts over visual evidence. |
| Approach: | They study the failure mode of large vision–language models by focusing on textual prompts over visual evidence. |
| Outcome: | The proposed model overestimates the number of objects in an image . it hallucinates additional waterlilies when asked to describe a mismatched number of items . the model ablation reduces prompt-induced hallucinosities by at least 40% without additional training . |
Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy (2026.findings-acl)
Copied to clipboard
Hosein Hasani, Mohammadali Banayeeanzade, Ali Nafisi, Sadegh Mohammadian, Fatemeh Askari, Mobin Bagherian, Amirmohammad Izadi, Mahdieh Soleymani Baghshah
| Challenge: | Large language models exhibit systematic limitations in counting tasks due to depth constraints. |
| Approach: | They propose a method that decomposes large counting tasks into smaller, independent sub-problems that the model can reliably solve. |
| Outcome: | The proposed method surpasses architectural limitations and achieves higher accuracy on large-scale counting tasks. |
SafeConstellations: Mitigating Over-Refusals in LLMs Through Task-Aware Representation Steering (2026.acl-long)
Copied to clipboard
| Challenge: | Current safety alignment methods fail to identify intended benign task before refusing to respond. |
| Approach: | They propose a method that uses inference-time trajectory-shifting to guide model behavior . they show that LLMs persist in refusing inputs containing harmful content . |
| Outcome: | The proposed approach reduces over-refusals with minimal impact on utility. |